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(54) Motion compensated image interpolation 



(57) The motion compensated temporal interpolator is capable of determining when a background scene is 
covered or uncovered by a foreground object by projecting motion vectors determined for the output field 
onto temporally adjacent input fields (i/pl, i/p2) and detecting the number of times each input pixel is used as 
a source for the output pixel - covering corresponds to multiple pixel use in the following field while 
uncovering corresponds to multiple pixel use in the preceding field. The motion vectors for output pixels in a 
covered area can then be corrected by forward projecting vectors from the preceding frame pair whereas 
output pixels in an uncovered area are corrected by backward projecting vectors from the following pair. 
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This invention relates to the. field of motion compensated image 
interpolation; 

Motion compensated image interpolation for purposes such as 
standards conversion is known, e.g. film to television, or from one 
television format to another. An example of such, a system is described 
in British Published Patent Application GB-A-2 231 7^9 (Sony 
Corporation). A problem" that occurs in these systems when there is 
relative motion between objects in the foreground , and objects in the 
background is that parts of the -objects in the background may be 
bovered or ioncovered by the .objects in ..the. foreground between 
successive input imsiges: ' 

An example of covering and uncovering of backgrpund objects is 
Shown in Figure 1 of the accompanying drawings. As the car in the 
foreground passes a stationary background, it uncovers the front wheel 
of the tow truck and covers the base of the radio mast. Covering and 
uncovering must be taken into account in the interpolation of an output 
field between the two fields shown in Figure 1. In,: particular, in the 
butput field, the radio mast and the tow truck wheel may be partially 
covered/uncovered, depending upon the temporal ppsition of, the output 
field. This has the result that only pixels from fi^^d 0 are available 
to generate the visible part of the radio mast.stand only pixels from 
field 1 are available to generate ;the visible ; part .pf the front wheel 

'of the tow truck. '. 

^ ^ It is therefdre necessary to detect the covering or uncovering of 
an object, in order that the interpolator can be steered only to use 
pixels/from the appropriate one of the input fields (i.e. field 0 for 
the radio mast and field 1 for the tow truck wheel in the above 
example). If this is not achieved or some other corrective measure 
taken, then a degradation in quality of the interpolated images will 
result. 

Viewed from one aspect the invention provides an apparatus for 
performing motion compensated image interpolation between temporally 
adjacent input arrays of input pixel values to generate an output array 
of output pixel values, said apparatus comprising: 

means for detecting a primary array of motion vectors associated 



with output pixel ^itions ai.d representing image mWn at said 
output pixel positions between said input arrays of input pixel values; 

means for projecting said motion vectors from said output pixel, 
positions to said input arrays of input pixel value?, to detect how many 
times each input pixel value is used as a source- for an output pixel 
value; and 

means for controlling subsequent . operation to . g^erate said 
output array of output pixel values in response to said detection of 
how many times each input pixel yalue is -^sed as. a so^rc^^.f^^ an output 
pixel value.' 

- Proiiection of - the motion vectors on t9 t^e input ,,ar pays of input 
pixel values to determine how.many times each input pixel value is used 
as a source for interpolation serves to detect areas of image that are 
being either covered or uncovered,. With this information, subsequent 
^ operation {e.g. interpolator steering, motion vector testing , and 
/■substitution, etc.) can be controlled in a variety of different ways so 
as to improve the interpolated image quality despite , the regions of 

cover and uncover. 

^ Whilst the means for controlling subsequent interpolation can 
take several different forms, in- preferred embodiments, said means for 

••■ controlling comprises: 

■ means for interpolating a test output array, of test output pixel 
lvalues using said primary array of motion, vectors; , 

r means for comparing said test output array, of test output pixel 
values with said input arrays of input pixel values to identify drop 
: out error positions at which corresponding pixel values at positions 
compensated for motion with said primary, array . of motion vectors 
projected from said input arrays of pixel values differ between said 
test output array of test pixel values and either of said input arrays 
of input pixel values and at which those input pixel values have both 

not been used as a source; 

means for detecting if primary motion vectors associated with 
: said drop out error positions project to positions in said input arrays 
of input pixel values at which those input pixel values have both not 
been used once as a source to identify confirmed drop out error 
positions; 

means for projecting primary, motion vectors at said confirmed 



drop out error pos^is to said primary array of moti^'ectors to 

identify erroneous" motion vectors ; 

means for selecting alternative motion vectors to replace said 
erroneous motion vectors..: to .produce .a: secondary array of motion 

vectors; and 

means for interpolating said output array of -output pixel values 
ijsing said secondary array of motion vectors. 

With this technique the control of subsequent interpolation takes 
the form of identifying erroneous motion vectors and replacing them 
with alternatives so as to produce a better quality interpolated image. 

. . An alternative preferred technique for controlling subsequent 
interpolation is one in which said means for controlling comprises: 

means for detecting pixels .source error positions in said input 
arrays of input pixel values at which input pixel values have been used 
"as a source more than once; ■:::■:-.<:■■■:,:■■'■ 

means for flagging output pixel positions coincident with said 

pixel source error positions; 

means for setting a flag, for those c output pixel positions 
surrounding each pixel soxorce - error position, within an error area with 
- dimensions proportional to that motion vector at said pixel source 
error position and temporal -displacement between said output array of 
output pixel values , and that input .array of input pixel values from 
which said flag arose; and . . .^-t; 

means for interpolating said output airay of output pixel values 
for flagged output pixel positions using one or more input pixel values 
from that input array of. input ■pixel values from which said flag did 

■not arise.- 

With this techtiique, the detection of regions of cover and 
uncover is used to steer the interpolator to use the appropriate input 
array of input pixel values to the exclusion of that in which the input 
pixel, values has been used as a source more than once. 

As a refinement to this technique to take account of errors in 
the motion vectors associated with regions of cover and uncover or for 
use in its own right, in preferred embodiments of the invention said 
means for controlling comprises: 

means for detecting pixel source error positions in said input 
arrays of input pixel values at which input pixel values have been used 



as a source more tn^^once; 

means for flagging output pixel positions, coincident with said 

pixel source error positions; and 

means for interpolating said output array of output pixel values 
for flagged output pixel positions using motion vectors at a 
corresponding position with said flagged output pixel positions 
associated with a temporally adjacent output array of output pixel 
values in a temporal direction of that , input ar.ray of input pixel 
values from which said flag did not arise. 

It will be . appreciated that detecting how many times each input 
pixel value has been used as a source is a computationally intensive 
activity. Whilst a .general purpose .computer, could perform this 
function, preferred embodiments incorporate special purpose hardware 
such that said means for projecting comprises: 

means for storing an array of marker values indicating how many 
time each input pixel value is used as a source for an output pixel 

value: . 
: means for generating a stream of motiCHi vectors and their 

associated output pixel position addresses; 

means for calculating an address .offset from each output pixel 
position to each of said temporally., adjacent input arrays of input 
pixel values by multiplying : that motion vector associated with said 
output pixel position by temporal . displacement to . said input arrays of 

input pixel values; :' 

. means for adding each of said address offsets to. said associated 
output pixel position address to yield source addresses in each of said 
temporally adjacent input arrays of input pixel values; and 

means for incrementing said marker values for said source 
addresses to indicate its use . as a source. 

Viewed from another . aspect the invention provides a method of 
performing motion compensated image interpolation between temporally 
adjacent input arrays of input pixel values to generate an output array 
of output pixel values, said method comprising the steps of: 

detecting a primary array of motion vectors associated with 
output pixel positions and representing image motion at said output 
pixel positions between said input arrays of input pixel values: 

projecting said motion vectors from said output pixel positions 



to said input array 
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input pixel values to detect hc^^any times 

eic^ i^puTpixe'l value is used as a. source for an. putput. pixel value; 

and ■' ■■ 

- controlling kubsequent operation to generate said output array of 
output pixel values in Wponse to said detection of how .nany ti«es 
each input pixel value is used as a source for an output pixel value. 

Embodiments of the invention will now be, described, by way of 
example only, with reference to the accompanying drawings in which: 

Figure i illustrates the problem of the cover and uncover of 

regions in adjacent images: 

Figures 2 and 3 illustrate steered interpolation using hi tboards 
(i.e. arrays indicating how many times each . pixel has been used as a 
source for interpolation); 

Figure 4 illustrates a circuit for generating hitboard data; 

Figure 5 illustrates a circuit for modified interpolation on the 

basis of hitboard data; 

Figures 6. 7 and 8, illustrate motion vector selection and steered 

interpolation on the basis of hitboard data; and 

■ ^^Figures 9 to 13. l4 to l8 and 19 to 23 illustrate the detection 
and replacement of erroneous inotion vectors utilising hitboard data. 

^ - Figure 1 illustrates two temporally adjacent fields that form 
input arrays of input image values between which it is desired to 
interpolate an output frame of output image values. The tow truck and 
radio mast are stationary in both field 0 and field 1. The car moves 
from in front of the tow truck in field 0 to in front of the radio mast 
in field 1. Thus, in field 1 the iCwer cab and front wheel of the tow 
truck has been uncovered whilst the lower portion of the radio mast has 
been covered. 

■ Depending upon the relative temporal position of the output frame 
to be interpolated, it may be necessary that the output frame contain 
a region of image that is either uncovered or covered in field 1. In 
order to be accurately interpolated, the uncovered region in the 
interpolated field should be derived solely from field 1 and 
conversely, the covered region should be interpolated entirely from 
field 0. In order to enable such steered interpolation yielding more 
accurate interpolation, regions that are covered and uncovered between 
temporally, adjacent frames must be detected. 



Figure 2 lli#ates. the use of ,a , hitboard. ,#nl,ue for 
detecting regions of cover and uncover. Field,0 and, fiel# 1 show a 
detecting s ,^„„„as (i e v • -3) behind a stationary 

line of letters moving leftwards (i.e. v 3J , , . . . 

,i e V . 0) bloclc .2. The output, fieia,«.bf , inte^lated i. at a 
position two thirds of the way -tow^ fi^d l (i.e. ^ ' 

in. field 0 the ri^tmost, letter visible is D. In field 1 the 
ri^t^st letter visible is- a., ,-In-.the.int,rpolated field the right»os^ 
litter that should be seen is F. It is. assumed that the correct sotion 
"t'rs have been established for all ,of the pixels in the - 
It.e. v . -3 for letters A to F and, v - 0 fof-, .^1 of the, pixels of the 

. ''""^ 'ulder^cnown operation With previous devices, each output pixel is 
derived by projecting its .notion vector forwards and backw^ to ^ 
tporally adjacent fields and using a su. of the pixel 
. ..put i.ages pointed to. each being weighted, according to «.e ra^ti^ 
te^oral proxisity of the output field to its input fx^l^s. In the 
^sent cLe. the letters E and F do not appear _in ,^ield 0 so »y 
contribution fro. field 0 would be lively to degrade,, the^ «age ,uali^ 
The hitboard for field 0. hitboard 0. is.deriyed by projecting 
bac^ards fro. the output field positions using each. of. the .otion 
vectors and incrementing a stored value for the associated Pixel in the 
field 0. All Of. the input pixel values, shown in field 0 are used as a 
source once apart fro, th* two input field .value, „at a»,left.hand edge 

of the block 2. ' \. 

Multiple hitboard entries in the temporally backward dxrectxon 
indicate a region of uncover. The control flags associated with the 
output field for the two pixels in which the double hits occur are set 
to "1" to indicate that interpolation should occur wholly from fxeld 1. 
in addition, the pixels on either side of those for, which the control 
.flags have been set are examined to see what motion vector xs 
.associated with them. I^e control flags are then extended to either 
Side by a distance given by the motion vector for the adjacent pixel 
multiplied by the temporal displacement from the field i.. which the 
double hit occurred. . . ,• , , r 

The remaining control flags where single hits occurred are set to 
. "b" to indicate that both field Q. and f ield . 1 should.be used as sources 
for interpolation. • . ' 
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horizon^^j^ and -yt,t and -^y^t pixels yeijt^ca^^f the set 

., control flag will also be set. ^ 

. , Figure 3 illustrates another example of the above technique. In 
this example, the line of letters is moving leftwards at v = -6 and the 
block 2 is moving rightwards at v = +3. Backwards projection from the 
output field results in the letters E to J projecting onto , the leftmost 
six pixels of the block 2 in field 0 in addition to the true projection 
from the leftmost six pixels of the block 2 in , the output field. 
Accordingly, these six pixels have multiple hitboard entries indicating 
that interpolation for them should occur only from field 1. In 
addition to the setting of those control flags directly corresponding 
to the multiple hitboard entries, the motion vectors associated with 
adjacent pixels p, q are assessed, to be -6 and +3 respectively. 
Multiplying each of these by the temporal displacement (.t=2/3) from the 
field 0 in which the multiple entry occurred extends the control flag 
,"1" by four pixel -; positions ^ -le^^^ pixel positions 

/.rightwards. : 

In this way, the pixels E to J -are correctly interpolated from 
ionly field 1 thereby, improving interpolated image quality. 

Figure 4 .-illustrates. -a .circuit for def-iying the entries to be 
made in the hitboards for temporally - adjacent forwards and backwards 
fields/frames . A pixel address counter 8 generates a sequence of pixel 
addresses moving through the output field to .be interpolated. In 
synchronism with the pixel address counter 8. the, motion vectors at the 
pixel addresses are supplied through input 10 to two multipliers 12, 
Ik. The output field to be interpolated is at a temporal position t 
between the temporally backward field 0 and the temporally forward 
field 1 . The temporal displacement from the output field to the field 
0 of "-t" is also applied to the multiplier 12 and a corresponding 
value "(l-t)" is applied to the other multiplier l4 to represent the 
temporal displacement from the output field to the field 1. 

The output from the multiplier 12 represents the offset to the 
pixel address that is produced by projecting the vector at the pixel 
position corresponding to the pixel address curr«itly being output from 
the pixel address counter 8 backwards onto field 0. Similarly, the 
output from the multiplier ik represents the corresponding offset to 



With the conlj^ flags thus, set, the output ima^l^ixel values 
for the letters E and F are derived wholly from field 1 with no 
contribution from field 0. This improves interpolated image quality- 

As can be seen from the above, the hitboards are employed to 
steer the interpolator, so that covered or uncovered parts of the image 
are derived from only the appropriate one of the input fields. The 
technique assumes that the correct motion vectors have been selected 

for each output pixel. 

An array of control flags (one per pixel of the output field) is 
set up, to indicate one of the following options; 

(a) interpolated from field 0 only; 

(b) interpolate from field 1 only; 

(c) interpolate from both output fields- : 
The flags are generated as follows; 

'1. Examine the two hitboards for double (or multiple) entries. 
If a more than once entry is found in hitboard 0, then only 
field 1 should be used for generating the output pixel at 
that position. Similarly; if a more than once entry is 
found in hitboard 1 . theii '^^only field 0 should be used to 
generate the output pixel at thiat ' position . The control 
■ ='-- f2,ags ■ tor those positionis are set atccordingly. 

• 2. - kbiere a control flag is set to indicate that only one field 
- ' -s^ be used in interpolation of that output pixels a 
number of adjacent control flag are set. At each edge of 
an area of set control flags,- set the n adjacent control 
flags to the same setting. Here, n is equal to the vector 
component at that edge in the direction of the adjacent 
flag (i.e. horizontally, vertically, up or down), 
miiltiplied by the temporal position (t) of the output 
field. Thus, if the horizontal vector component at the 
left edge is x^, the horizontal vector component at the 
right edge is x^., the vertical vector component at the top 
.edge is y^, the vertical vector component at the bottom is 
and the temporal position is t (where 0 ^ t ^ 1) , then 
the control flags that are within -x^t and +x^t pixels 



the pixel address the projection forwards onto 

address offsets are supplied to respective adders l6 and l8 where they 
are summed with the current pixel address from the pixel address 
counter 18 to yield source addresses for projected interpolation in the 
backward and forward fields. These source addresses are applied to the 
hitboards .20 and 22 ^(arrays of RAM with a storage location for each 
pixel position) where they cause the value corresponding the address 
pointed to be incremented to indicate its intended luse as a source of 
interpolation. 

Figure 5 illustrates a circuit for controlling interpolation 
utilising hitboards- Vectors generated in the known manner are 
supplied to a vector selector 24 from which they pass to a vector 
buffer 26 and a hitboard generator 28. The hitboard generator 28 has 
the form described in relation to Figure 4. The vector buffer 26 
serves to delay "the vectors for a period corresponding to the 
processing lag >n the tipper arm of the circuit of Figure 5- The 
hitboards 20 and 22 for respective backwards and forwards projections 
from the output field in conjunction with the multiple entry detector 
30 serve to detect areas of cover and uncoyer/in the image. The 
multiple entry detector 30 supplies signals indicative of multiple 
• entries to a control flag generator 32 that serves to produce the 
•control flags that steer the interpolator and also performs the edge 
extension of the control flag discussed above. The control flag 
generator 32 is supplied with the delayed vectors from the vector 
buffer and the temporal position of the output field in addition to the 
output from the multiple entry detector 30 as this information is 
heeded to generate the control flags to the interpolator (downstream) 
that control whether field 0. field 1 or both fields are used as the 
source of interpolation. 

Figure 6 illustrates the use of hitboard entries to control 
subsequent interpolation by forcing the use of motion vectors from the 
previous or next output fields in the interpolation of covered and 
uncovered objects respectively. The sequence of input frames i/pl» 
i/p2 and i/p3 show a block 34 moving rightwards against a background 36 
that is moving leftwards. ' Output frames o/pl, o/p2 are to be 
interpolated at positions between the input, frames. The pixel 
indicated by an "o" are uncovered by the ' divergence between the 



respective scenes do not have valid motion vectors l^R)ciated with 
them . : 

The default motion vector associated with the iincovered pixels a 
is zero. This leads to multiple entries in the backward hitboard to 
input field i/pl at the points indicated by an "x". The other 
uncovered pixels are at positions where the motion vectors are derived 
from the moving block and at which no multiple entries in the backwards 
hitboard occur. 

The multiple entries in the backwards hitboard at the positions 
"x" indicate divergence and that the motion veqtors should be taken 
from the succeeding frame pair i/p2 and i/p3- These motion vectors 38 
can then .be used in the interpolation of the points 40 for which motion 
vectors had not previously been correctly identified. The 
interpolation may then proceed using these corrected motion vectors 
according to the technique illustrated in Figixres 2 and 3- 

In a similar manner, the pixels b are -uncovered between input 
frames i/p2 and i/p3 and yield multiple entries in the backwards 
hitboard. Accordingly, the motion vectors associated with the 
succeeding input frames are used for the. pixels 42 interpolated in the 
output field o/p2. 

Figure 7 illustrates the situation for convergence. In this 
case, the block 34 and the background 36 are converging at the right 
hand edge of the block 34. The pixel positions c in the output frame 
o/pl are hidden by the convergence with the result that they have a 
default zero motion vector that yields a double hitboard entry in the 
forward direction corresponding to input field i/p2. The forward 
direction indicates that the vectors for the points should be taken 
from the backwards direction, i.e. from vectors derived between input 
fields i/pO (not shown) and i/pl- As before, these corrected motion 
. vectors can then be used for interpolation. 

:: In a similar manner to the above, the motion vectors for the 
pixels d can be taken from those between input frames i/pl and i/p2 in 
view of the double hitboard entry in the forward direction. 

: The above . operation can be considered as follows: 

1. ; < Converging vectdirs (i.e. multiple writes) in the backward 
direction indicate a diverging scene, hence vectors will 
= :have to be obtained from the next choice of output vectors. 
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OR diverg^i^ vectors (i.e. zero writes) in^Pe forward 
direction indicate a diverging scene, hence vectors will 
have to be obtained from the next choice of output vectors. 



2. Diverging vectors (i.e. zero writes) in the backward 
direction indicate a converging scene, hence vectors will 
have to bfe obtained -from the previous choice of output 
vectors. 

- OR converging vectors (i.e. multiple writes) in the forward 
• direction Indicate a converging scene, hence vectors will 
have to be obtained from the previous choice of output 
vectors. 

Figure 8 illustrates a sequence of input fields i/pl to i/p6. In 
this sequence a dark block starts and then stops moving against a 
background whilst a camera tries to track it. Between frames i/p2 and 
i/p3 the block moves and whilst the camera tries to follow it 
(indicated by moving background) . the block still moves in the frame. 
This is also the case between frames i/p3 and i/p4. Between frames 
i/p4 and i/p5 the camera is successfully following the block holding it 
stationary within the frame whilst the background moves. Between 
frames i/p5 and i/p6 the block stops moving. Output fields o/pl. o/p2 
and b/p3 are at positions between the input fields. 

■ In this example, it will be seen that when a poor match occurs in 
vector selection coincident with zero writes in the backward write 
projection, then the vectors used should be those used in the previous 
output field forward projected. 

■ In Figure 8, it can be seen that the vectors chosen in this way 
are a, b, c. d. i and j. The vectors a. b. c and d are in fact the 
wrong vectors, but represent the best "available" vector in the 
circumstances. The vectors i and j are the correct vectors. 

Conversely, when a poor match occurs in vector selection 
coincident with zero writeis in the forward write projection then the 
vectors used should be those used, in the next output field backward 
projected. In this case these vectors are k, 1, m, n, o and p. which 
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in each case are correct vectors, .. .This is also^Ps case for 

vectprs e, f . g and h. 

Figures 9 to 13 illustrate a third technique in which hitboard 
entries are used to modify subsequent interpolation so as to improve 
output image quality. In this case, the hitboards are used, with other 
information, to correct erroneously selected motion vectors. 

The processing takes place .as part .of the interpolation of the 
output field. A test interpolation is first performed, and the motion 
vectors and interpolator steering modified (as necessary) for final 
interpolation. Hitboards (backwards/ forwards write flags) are set up 
indicating the intersection of the selected motion vector for each 
output pixel with the two input fields. In addition, the output frame 
generated in the test interpolation is projected onto the input frames 
that generated it by running all the motion vectors backwards (i.e. a 
sort of reverse interpolation). If the test interpolation was 
accurate, then this projection should regenerate the input frames (this 
comparison will be referred to as forward and backward, projection) . 
^ The reverse projected frames.are subtracted, on a pixel by pixel basis, 
from the true input frames to. produce error . fraB^es. If the 
interpolation was . completely , accurate., then all the pixels in the error 
frames would be rrzero. Positions of inaccurate interpolation are 
indicated by non-zero, values in the error frames. 

As shown in Figures 9 and 10 a pixel 38 moves horizontally by 
four pixels between input, frame (n), and input frame (n+1). The motion 
•vectors selected by . conventional means are shown in the array ^0. The 
: two input frames and the motion vectors are then used to generate a 
test output frame 42 and to generate the forward and backward hitboards 
and 46 respectively. It will be seen that for the middle pixel 
three positions from the left in Figure 9. a match of a white pixel to 
a white pixel with zero motion is just as good an apparent match as the 
true black to black pixel match for the moving pixel 38. 

Forward and backward projection is then performed by subtracting 
each output pixel in the reverse projection from the corresponding 
position in input frames (n) and (n+1) to generate the forward and 
backward error frames. Differences, between the input pixels and the 
forward and backward projected pixels are marked 48. 50 as error 
positions in the input frame cqprdinates (the differences, are compiared 



with a threshold val^Jbifferences exceeding the threshol^Rlue being 
marked as error positions, filtering voi^d also be -us^d to reduce the 
effect of objects and object edges that fall partially in one. pixel and 
partially in it neighbour). • Pixels for which two. or more entries are 
present in the hitboards are ignored (masked) .in, this error niarking as 
is indicated by the 52. 

The motion vectors are then corrected, .by .following set of 

rules: ■ 

1. Examine the hitboards. looking for a pixel position having 
zero entries in the forward and backward hitboards. 

2. Does a discrepancy in the forwai>d or backward error frames 
occur at this point (i.e. an error position)? 

3. Does the vector used at this position point to a multiple 
hi tboard entry? 

k. If the answers to (2)-and. (3) are yes. then, a vector can be 
identified that is probably erroneous. That vector can be 
discarded and a neighbouring (different) vector tried. The 
position -of . .the erppnepus vector 

if the present vector is positive and, t< 0.5. then 
the position of the jerronepus. vector is d?.splaced by +(t.v) 
with respect to the ^erO: write . fliag in the backward 
-hi tboard; ■■■■V- = 

: . if . the present vector is positive and t > 0.5. then 
the position of the erroneous vector is displaced by -((1- 
. t).v) with respect.to :the, zero write. fla^ forward 
• ■• -hi tboard i 

Returning now to Figure 10. . this illustrates the test 
interpolation and hitboard generation for the image regions shown in 
Figure 9. In particular the numbers of times each pixel is used as a 
source is recorded in the hitboards as indicated by the numbers 5*^. It 
will be seen that the pixel 38 is not used as a source in either the 
forward or reverse input frames -and thus the interpolated output frame 
does not contain this dark pixel 38. Accordingly, an error position 
will be detected in the backward and forward error frames at the 
positions corresponding to the dark pixel 38 in the input frames n and 
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As shown in Figure -11 , the error position 48 from the backward 
error frame in Figure 9 is searched in the hitboards and 46 to see 
if it corresponds to a position not used as a source- In this case it 
5 does so correspond and accordingly an erroneous vector is probably 

present. Since this error is detected in the backward error frame, 
there is no need to correct for the same error in the forward 
projection. The position of this vector is calculated as t.v from the 
position in the backward direction, .i.e. in this case one pixel 
10 displacement to the right at position 56. The old vector at this 

position (a: zero motion vector) is replaced with an alternative vector 
chosen from nearby in accordance with usual vector, selection practice 

(in this case v) . 

Figures 12 and 13 illustrate what would occur when this test is 

15 performed with the corrected iotion vectors, ;i-e. a situation in which 

no errors are present. As will be seen, the pixel 38 in this case 
appears in the output frameiV -- Further, the backward and forward error 
frames do . not contain any error positions, i.e. pixels that do not 
project from their input frame to a corresponding pixel in the output 

20 "frame/'" 

■ Figures 14 to I8 illustrate another example of the third 
technique in which a dark pixel 58 moves behind a stationary dark area 
60, i.e. convergence. With reference to Figure l4. the pixel 58 moves 
horizontally by four pixels between input frame (n) and input frame 

25 (n+1). The motion vectors selected by conventional means are shown in 

the array 62. The two input frames and the motion vectors are then 
used to generate a test output frame 64 and to generate the forward and 
. backward hitboards 66 and 68 respectively. 

Forwards and backwards projection is performed and then these 

30 frames are compared with the corresponding input frames to generate 

forward and backward error frames . Differences between the input 
pixels and the forward and backward projected pixels are marked 70 as 
error positions in the input frame coordinates (the differences are 
compared with a threshold value , differences exceeding the threshold 

35 value being marked as error positions; filtering would also be used to 

reduce the effect of objects and object edges that fall partially in" 
one pixel and p£u?tially in it neighbour) . Pixels for which two or more 



:^^he hitboards are ignored { masked )^r.t 



entries are present^W^he hitboards are ignored { masked )^r.this error 
marking as is indicated by the 72- 

The motion vectors are then corrected by the set of rules as 
given above. 

Figure 15, this illustrates the test interpolation and hitboard 
generation for the image regions shown in Figure 14. In particular the 
numbers of times each pixel is used as a source is recorded in the 
hitboards as indicated by the numbers 7^- It will be seen that the 
pixel 58 is not used as a source in either the forward or reverse input 
frames and thus the interpolated output frame does not contain this 
dark pixel 58, Accordingly, an error position is detected in the 
backward error frame at the position cor^responding to the, dark pixel 58 
in the input frame n. 

As shown in Figure I6, the error position 7O from the backward 
error frame of Figure l4 is searched in the hitboards 66 and 68 to see 
if it corresponds to a position not used as a source. In this case it 
does so correspond and accordingly an erroneous vector is probably 
present. The position of ^this vector is. . calculated as. t-v from the 
posi t ion in the backward direction , i • e . ? in this case one pixel 
displacement to 'the^ right 'at position 77-^ The old . vector at this 
position (a zero motion vector), is replaced with-an alternative vector 
chosen from nearby in accordance with usual vector ^selection practice 
^Xih 'this case v). "'■*•> -o/" ; 

Figures 17 and I8 illustrate what would occur when this test is 
performed with the corrected rmotion vectors, ive. a situation in whiqh 
no errors are present. As will be seen, the pixel 58 in this case 
appears in the output frame. Further, the backward and forward error 
frames do not contain any error positions, i.e. pixels that do not 
prbject-from their input frame to a corresponding pixel in the output 
'frame.^'--- 

Figures 19 to 23 illustrate another example of the third 
technique in which a dark pixel 78 moves out from behind a dark area 80 
(divergence). With reference to Figure 19. the dark pixel 78 moves 
horizontally by four pixels between input frame (n) and input frame 
(n+1) . The motion vectors selected by conventional means are shown in 
the array 82. The two input frames and the motion vectors are then 
used to generate a test output frame 84 and to generate the forward and 



backward hitboards ^^and 88 respectively- 

Forward and backward projection is then performed by subtracting 
each output pixel in the reverse projection from the corresponding 
position in input frames (n) and (n+l) to generate .the forward and 
backward error frames . Differences between the input pixels and the 
forward and backward projected pixels are marked 90 as error positions 
in the inpiit frame coordinates (the differences are compared with a 
threshold value, differences exceeding the threshold vaiue being marked 
as error positions; filtering would also be used to reduce the effect 
of objects and object edges that fall partially in one pixel and 
partially in it neighbour) • Pixels for which two or more entries are 
present in the hitboards are ignored (masked). iJi this error marking as 

is indicated by the 92. 

The motion vectors are then corrected by .the s^t of rules given 

above. ' '■"'"^ ■ 

' Figure 20 illustrates the test interpolation and hitboard 
generation for the image regions shown in Figure 19- In particular the 
numbers of times each pixel is used as r a source is recorded in the 
hitboards as indicated by: the numbers =94. It wrill be seen that the 
pixel 78 is not used as a source in either the forward or reverse input 
frames and thus the interpolated output frame .does not contain this 
dark pixel 78 . Accordingly , an terror position will be detected in the 
forward error frame at the position corresponding to. the dark pixel 78 
in the input frame (n+l) . - /•;: 

As shown in Figure 21, the - error position 90 from the forward 
error frame in Figure 19 is searched in the hitboards 86 and 88 to see 
if it corresponds to a position not used as a source. In this case it 
does so correspond and accordingly an erroneous vector is probably 
present. Since this error is detected in the forward error frame 86, 
there is no need to correct for the same error in the backward error 
frame. The position of this vector is calculated: as -(1-t) .v from the 
position in the forward direction, i^e. in this case one pixel 
displacement to the left at position 98. The old vector at this 
position (a zero motion vector) is replaced with an alternative vector 
chosen from nearby in accordance with usual vector, selection, practice 
(in this case v) . 

Figures 22 and 23 illustrate what would occur when this test is 
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performed with the ^j^ected motion vectors, i.e. a sitdl^Rn in which 
no errors are present. As will be seen, the pixel 78 in this case 
appears in the output frame. . Further, the backward and forward error 
frames do not contain any error positions, i.e. pixels tiiat do not 
project from their input frame to a corresponding pixel, in the output 
frame. 
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CLAIMS 



1. Apparatus for performing motion compensated image interpolation 
between temporally adjacent input arrays of input pixel values to 
generate an output array of output pixel values, said apparatus 
comprising: 

means for detecting a primary array of motion vectors associated 
with output pixel positions and representing image motion at said 
•output pixel positions ijetween said input arrays of input pixel values; . 

means for projecting said motion vectors from said output pixel 
positions to said input arrays of input pixel values to detect how many 
times each input pixel value is used as a source for an output pixel 
value; and 

means for controlling subsequent operation to generate said 
output array of output pixel values in response to said detection of 
how many times each input pixel value is used as a source for an output 
pixel value. • 

2- Apparatus as claimed in claim 1, wherein said means for 
controlling comprises : 

means for interpolating a test output array of test output pixel 
values uising said primary array of motion vectors; 

means for comparing said test output array of test output pixel 
values with said input arrays of input pixel values to identify drop 
out error positions at which corresponding pixel values at positions 
compensated for motion with said primary array of motion vectors 
projected from said input arrays of input pixel values differ between 
said test output array of test pixel values and either of said input 
arrays of input pixel values and at which those input pixel values have 
both not been used as a source; 

means for detecting if primary motion vectors associated with 
said drop out error positions project to positions in said input arrays 
of input pixel values at which those input pixel values have both not 
been used once as a source to identify confirmed drop out error 
positions; 

means for projecting primary motion vectors at said confirmed 
drop put error positions to said primary array of motion vectors to 
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identify erroneous flBFon vectors; 

means for selecting alternative motion vectors to replace said 
erroneous motion vectors to produce a secondary, array of motion 
vectors; and 

means for interpolating said output array of output pixel values 
using said secondary array of motion vectors. 

3. ■ Apparatus as claimed in claim 1. wherein said means for 
■ controlling comprises : 

'■■"means for detecting pixel source error positions in said input 
arrays of input pixel values at which , input pixel values have been used 
as a source more than bhce; 

" means for flagging butput pixel positions coincident with said 
pixel source error positions ; 

■ means for setting a flag for those output pixel positions 
surrounding each pixel source error position within an error area with 
dimensions proportional to that motion vector, at said -pixel source 
error position and temporal displacement between said output array of 
output pixel values and that input : array -of input pixel values from 
which said flag arose; and 

means for interpolating said output array of output pixel values 
for flagged output pixel positions using one or more input pixel values 
from that input array of input pixel ■ values , from which said flag did 
not arise. 

4. Apparatiis as claimed in claim 1. wherein: said means for 
bontroiiing comprises: 

- means for detecting pixel source error positions in said input 
' arrays of input pixel values at which input pixel values have been used 
as a source more than once; 

means for flagging output pixel positions coincident with said 
pixel source error positions; and 

' means for interpolating said output array of output pixel values 
for flagged output pixel positions using motion vectors at a 
corresponding position to said flagged output pixel positions 
associated with a temporally adjacent output array of output pixel 
values in a temporal direction of that input array, of input pixel 
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values from which flag did not arise. 

5. Apparatus as claimed in any one of the preceding; claims * wherein 
said means for projecting comprises: 

means for storing an array of marker values indicating how many 
time each input pixel value is used as a source for an output pixel 
value; . 

means for generating a stream of motion, vectors and their 
associated output pixel position addresses; 

means for calculating an address offset from each output pixel 
position to each of said temporally adjacent input arrays of input 
pixel values by multiplying that motion vector associated with said 
output pixel position by temporal displacement to said input arrays of 
• input pixel values; 

means for adding each of said address offsets to said associated 
output pixel position address to yield source addresses in each of said 
temporally adjacent input arrays of input pixel values; and 

means for incrementing said marker values , for , said source 
addresses .to indicate its use as a source. ; = 

6. A method of performing motion compensated image interpolation 
between temporally adjacent .input arrays of input pixel values to 
generate an output array of output pixel values, said met^Ojd. comprising 
the steps of: 

detecting a primary array of motion vectors associated with 
output pixel positions and representing image motion at said output 
pixel positions between said input arrays of input pixel values; 

projecting said motion vectors from said: output pixel positions 
to said input arrays of input pixel values to detect, how many times 
each input pixel value is used as a source for an output pixel value; 
and ■ ■ ■ ■ ■ 

controlling subsequent operation to generate said output array of 
output pixel values in response to said detection of how many times 
each input pixel value is used as a source for an output pixel value. 




7. A method as claimed in claim 6, wherein said step of controlling 
comprises the steps of: 
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interpolating ijptest output array of test output^fcel values 
using said primary array of motion vectors; 

comparing said test output array of test output pixel values with 
said input arrays of input pixel values to identify drop out error 
positions at which coincident pixel values differ between said test 
output array of test pixel values and either of said input arrays of 
input pixel values and at which those input pixel values have both not 
been vised as a source ; 

' " ■ dfeti^cting if primary motion vectors associated with said drop out 
"etror i)ositidns i>roject to positions in said input arrays of input 
•pixel values at which those input pixel values have both not been used 
onc^ as a sourc:e to identify confirmed drop out error positions; 

projecting ^primary motion vectors at said confirmed drop out 
error ' positions to said primary array of - motion - vectors to identify 

erroneous motion vectors; 

selecting alternative motion vectors to replace said erroneous 
motion vectors to produce a secondary array of motion vectors; and 

interpolating said output array of output pixel values using said 
' secondary array of motion vectors 

8/ A method as Claimed in claim 6, wherein risaid step of controlling 
comprises the st0ps of : 

" ' ^detecting pixel source error positions in said input arrays of 
input pixel values at which input pixel values have been used as a 
source more than once; 

~ flagging output pixel positions coincident with said pixel source 
error positions; 

setting a flag for those output pixel positions surrounding each 
pixel source error position within an error area with dimensions 
proportional to that motion vector at said pixel source error position 
and temporal displacement between^ said output array of output pixel 
values and that input array of input pixel values from which said flag 
arose; and 

interpolating said output array of output pixel values for 
flagged output pixel positionis using one or more input pixel values 
from that input array of input pixel values from which said flag did 
not arise. 



^ll^med In claim .6., wherein s^ald ,ste|^IP( 



9- A method as ^^^med In claim .6., wherein said stepVF controlling 
comprises the steps of: 

detecting pixel source error positions in said input arrays of 
input pixel values at which . input ; pixel values have been used as a 
5 isource more than once; 

flagging output pixel; ppsitioxis. coincideint with said pixel source 
error positions; and 

interpolating said output array ...of;, output ..p^^ values for 
flagged output pixel positions,, using motion yectors^^ a coincident 
10 position with said flagged output pixel positions ass^^^ with a 

temporally adjacent output array of output pixel y^l^ a temporal 

direction of that input array of inptit pixel, yalues from which said 
flag did not arise interpolating uses motion .yecto a corresponding 

position with said flagged output, pixel positions , associated with a 
15 temporally adjacent output array of output pixel va3.ue$ in a temporal 
direction of that input array of yi^nput p^^ from which said 

flag did -not arise.,-'- v.:^: :V 

10. A method as claimed in any one :Qf claiips.:6 tp.9j wherein isaid 
20 step of projecting comprises the steps of: 

storing an array of marker values indicating how many time each 
input pixel value is used as a source for. an output pixel value; 

generating a stream of motion vectors, ai^d thei output 
pixel position addresses; 
25 calculating an address offset from each output pixel position to 

each of said temporally adjacent input arrays of input pixel values by 
multiplying that motion vector associated with said output pixel 
position by temporal displacement to said i input arrays of input pixel 
. -values; • . 
30 adding each of said address of f sets to said associated output 

pixel position address to yield source addresses in each of said 
temporally adjacent input arrays of input pixel values; and 

incrementing said marker values for said source addresses to 
Indicate its use as a source* 
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11. Apparatus for performing motion compensated image interpolation 
substantially as hereinbefore described with reference to the 
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accompanying drawini 
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.12. A method of performing motion compensated image interpolation 
substantially as hereinbefore . described witli reference to. the 
5 accompanying drawings. 
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